1. Field of the Invention
The present invention relates to computerized simulation of hydrocarbon reservoirs in the earth with high performance computing (HPC) clusters, and in particular with scalable and expandable HPC clusters which have sub-clusters of different generations of processors.
2. Description of the Related Art
In the oil and gas industries, massive amounts of data are required to be processed for computerized simulation, modeling and analysis for exploration and production purposes. For example, the development of underground hydrocarbon reservoirs typically includes development and analysis of computer simulation models of the reservoir. These underground hydrocarbon reservoirs are typically complex rock formations which contain both a petroleum fluid mixture and water. The reservoir fluid content usually exists in two or more fluid phases. The petroleum mixture in reservoir fluids is produced by wells drilled into and completed in these rock formations.
A geologically realistic model of the reservoir, and the presence of its fluids, also helps in forecasting the optimal future oil and gas recovery from hydrocarbon reservoirs. Oil and gas companies have come to depend on geological models as an important tool to enhance the ability to exploit a petroleum reserve.
In simulation models, the reservoir is organized into a number of individual cells. Seismic data with increasing accuracy has permitted the cells to be on the order of 25 meters areal (x and y axis) intervals. For what are known as giant reservoirs, the number of cells is the least hundreds of millions, and reservoirs of what is known as giga-cell size (a billion cells or more) are encountered.
One type of computer system which has been available for processing the vast amounts of data of the types encountered in reservoir simulation has been high performance computing (HPC) grids. An HPC grid system takes the form of a group of powerful workstations or servers, joined together as a network to function as one supercomputer.
U.S. Pat. No. 7,526,418, which is owned by the assignee of the present application, relates to a simulator for giant hydrocarbon reservoirs composed of a massive number of cells. The simulator mainly used high performance computers (HPC). Communication between the cluster computers was performed according to conventional, standard methods, such as MPI mentioned above and Open MP.
High Performance Computing (HPC) grids typically have been made available for three years replacement cycles for their computer hardware from the supplying HPC manufacturer. Typically, a new HPC computer system designed for running reservoir simulation has been is bought every year either as a replacement for an older system, or as additional growth in compute requirements to run larger models. HPC data centers with such replacement cycles thus typically have at least three generations of computer hardware available for use. These existing systems consume space, power and cooling. They also require maintenance support contracts. It is expected that these systems be utilized efficiently.
Generational differences between these systems have followed Moore's law where the number of transistors, and thus performance, doubled approximately every eighteen months to two years. The difference in performance and speed between first generation and third generation hardware of an installed HPC grid available on the floor is typically on the order of three to four times.
Users tend to demand the newer faster systems (also known as sub-clusters) leaving older ones severely underutilized. These generational sub-clusters are connected together in a grid fashion allowing simulation jobs to straddle multiple sub-clusters. In reality, these sub-clusters are used in a stand-alone fashion because allocation of compute resources across multiple generations of hardware slows down simulation jobs to the slowest hardware in the allocation.
The current mode of running reservoir simulation jobs on the HPC environment is by allocating HPC sub-clusters for the users' runs. These physical clusters cannot be altered once built, due to the physical wiring involved between the compute nodes. Furthermore, the reservoir simulation software assumes equal workload sharing and homogeneous type of CPU's (i.e. same speed) when distributing the load between the compute nodes, otherwise the simulator will perform based on the slowest CPU in the cluster if they are different. This has prevented the running of larger simulation models on grid computers, and also prevented optimal utilization of heterogeneous physical machines when interconnected together.